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Cross Reference to Related Applications 

[001] This application claims the benefit of the fiHng date of the copending U.S. Provisional 
Patent AppUcations Serial No. 60/463,273 entitled "Three dimensional tangible interface for 
interacting with spatial-temporal data using a laser scanner" filed on April 16, 2003 and Serial No, 
60/463,357 entitled 'Three dimensional tangible interface for interacting with spatial-temporal data 
using infrared hght sources and infrared detectors" filed on April 17, 2003, the disclosures of which 
are hereby incorporated herein by reference. 

Field of the invention 

[002] This invention relates to systems for visualizing three dimensional datasets. 

Summary of the invention 

[003] In its preferred form, the present invention enables a user to intuitively explore and 
visualize complex three dimensional datasets by manipulating the position and shape of a physical 
modeUng surface. 

[004] By alerting the shape of the physical material, the user selects the geometry of a free 
form surface that is used as a cutting surface to intersect a voxel dataset. The intersected voxel 
values are then converted to image data that is projected back onto the surface of the physical 
material. The system consists of a manually manipulatable physical model that defines a surface 
whose shape or position may be altered, a position sensor for generating position data specifying 
the geometry of the surface, a processor for comparing the voxel dataset with the sensed position 
data to identify those voxel values which have positions in the array that correspond to the 
geometry of the surface, and a projector for illuminating the surface with an image representative of 
the voxel data at the intersection of the array and the defined surface geometry. 

[005] The physical object used to model the surface geometry is constructed of a material 
which forms a surface whose shape and position varies when said object is manually manipulated 
and upon which an image may be projected and viewed by a user. The material may comprise a 
deformable material, such as clay or plastic, which defines a manipulatable curvilinear surface 
whose three dimensional geometry is captured by a laser scanner or the like. Altematively, the 
material may consist of an aggregation of smaller movable objects, such as beads or rectilinear 
blocks, which may be individually moved to alter the shape or position of the surface. The material 
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may be translucent and the geometry of the surface may be detected by a camera which captures 
the intensity of the light passing through the material to the surface such that the surface light 
intensity varies in relation to the depth of material. 

[006] The data evaluation methods and apparatus contemplated by the invention provide the 
benefits of both physical and digital data representations. The interface shares many of the 
advantages of a standard graphical user interface (GUI) by allowing colored, layered, annotated 
and/or dynamic information to be represented to the user. As with GUI interfaces, different layers 
of information such as tissue types, solid objects or annotations can be invoked at will, offering a 
level of interactivity that is not possible using non-augmented three dimensional models. However, 
unlike GUI methods which present three dimensional data in two dimensional cross-sections or, at 
best, perspective renderings, the present invention presents three dimensional data in three 
dimensional space, allowing users to gain an intuitive understanding of complex relationships 
through the natural mechanisms of human spatial understanding (bioptical vision, position parallax 
and the tactile senses). These benefits are provided with a relatively simple and inexpensive 
hardware, without the need of tethering technologies such as tags, position-tracking devices, gloves 
or headgear as commonly used in virtual reahty or artificial reaHty environments. 

[007] The interface allows users to take advantage of the natural dexterity of their hands to 
quickly define cutting surface geometries of arbitrary complexity. These same geometries might 
take hours to define with the same level of control using traditional GUI methods. The interactive 
components of the system allow users to explore spatial relationships through the manipulation of a 
physical material. As with conventional physical models, this approach gives a tangible immediacy 
to three dimensional data representations with the additional benefit of a 1 : 1 mapping of scale 
where appropriate. However, unlike conventional physical models, which are normally rigid and 
offer limited scope for interaction, the invention allows users to cut a section at any desired location 
in a voxel dataset and to visualize the corresponding values in the array of volumetric data. 

[008] These and other features and advantages of the present invention may be more clearly 
understood by considering the following detailed description of specific embodiments of the 
invention. In the course of this description, fi-equent reference will be made to the attached 
drawings. 
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Brief description of the drawings 

[009] Fig.l is a block diagram illustrating a first embodiment of the invention employing a 
laser scanner to detect the geometric shape and position of a surface defined by a deformable 
modeling material, and 

[010] Fig. 2 is a block diagram of a second embodiment of the invention employing an 
infi-ared camera to capture the geometry of a surface formed by a bed of glass beads illuminated 
fi-om below by an array of LEDs. 

Detailed description 

[Oil] The system contemplated by the present invention is called a "phoxel space interface" 
and permits a user to physically interact with voxel data represented as "phoxels" (physical 
volumetric pixels). As used herein, a "voxel" (volumetric pixel) is the smallest discrete spatial 
component of a digital "volume" that provides a three dimensional representation of information. A 
"phoxel" is a voxel that is represented visually on a physical surface in three dimensional space. 
The system contemplated by the invention uses ttie surface geometry of a physical material to 
define a cross section surface that passes through the volume described by the voxel dataset. The 
values in the dataset that are intersected by this surface geometry are converted to visual images 
that are projected back onto the surface of the physical material. The system enables users to easily 
define the location and geometry of a desired cross section through the voxel dataset and to 
interactively view the results. The system includes the following components: 

[012] (a) Physical modeling material. This material is used to define the geometry of a 
surface which intersects the dataset. As described in more detail below, different kinds of materials 
(such as clay, plasticine, glass beads, cubic blocks, etc.) can be used and the particular material is 
chosen depending on the type of voxel data to be explored and the manner in which the position of 
the intersecting surface geometry is measured. The material chosen must also provide a suitable 
surface upon which an image representing the phoxel data can be projected. Plastic materials such 
as clay or plasticine that are easily shaped and, after being shaped, retain their shape, may be used 
to define a surface whose position is determined by a laser scanner. The physical modeling material 
may also consist of a glass beads which are individually movable, like the grains of sand in a 
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sandbox, to define a surface whose geometry may be determined by optical measurements. 
Individually movable rectilinear blocks may be stacked to define a surface whose position may be 
determined using a laser scanner or optical measurement. 

[013] (b) A voxel dataset. This dataset is the source of information to be explored using the 
phoxel-space interface. The voxel dataset comprises an array of point values organized in a regular 
3-dimensional grid. Voxel datasets may be generated using sensing technologies such as NMR 
(nuclear magnetic resonance) or CTI (computerized thermal imaging), or by computational 
simulations such a CFD (computational fluid dynamic) simulations. Note that the voxel dataset 
may be dynamically varying, resulting in a time varying image being projected onto the physical 
model surface that reflects time- varying changes at the surface / voxel volume intersection. 

[014] (c) A three-dimensional sensor. A thee dimensional sensor is required in order to 
capture the surface geometry of the physical modeling material. This sensor is preferably capable 
of capturing changes in the siu-face geometry in real-time, thereby allowing interactive exploration 
of the given voxel dataset by the user. The type of sensor selected may be dependent on the kinds 
of modeling material used as explained below. 

[015] (d) A data processor. A conventional computer may be used to store the voxel dataset, 
control the three-dimensional sensor to produce data indicating the current position of the physical 
surface, process the three-dimensional sensor output and identify the subset of voxel values that are 
intersected by the physical surface geometry. 

[016] (e) An image projector. The image projector, which may take the form of a 
conventional LCD, CRT or DLP video projector of the type used to display computer generated 
graphical data, is employed to project RGB (or gray-scale) pixel data representative of the voxel 
values rendered by the processor back onto the surface of the physical modeling material. The 
video projector is calibrated with the three- dimensional sensor to allow projected voxel values to 
correspond with points on intersection on the surface geometry. The type of projector used is 
partially dependent on the 3-dimensional sensing technique as described below. 

[017] Two illustrative implementations of the invention are presented in the description that 
follows. The first embodiment, seen in Fig. 1, employs a deformable plastic material to define the 
three dimensional surface and includes a laser scanner to determine the position of the surface. The 
second embodiment, illustrated in Fig. 2, uses a bed of translucent beads to create a physical three 
dimensional surface which can be formed by the user and a camera to measure the intensity of light 
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transmitted upwardly through the beads to determine the depth of the beads to capture the geometry 
of the surface. The laser scanner embodiment of Fig. 1 is highly reliable and accurate, but more 
expensive to construct than the camera embodiment of Fig. 2. 

[018] The arrangement shown in Fig. 1 uses a commercially available triangulation based 
laser scanner 103 such as the Minolta Vivid-900 available from the Instrument Systems Division of 
MINOLTA USA, 101 Williams Drive, Ramsey, New Jersey to capture the surface geometry of the 
physical modeling material. The laser scanner is calibrated with a video projector 105 which is 
positioned with the laser scanner 103 above the physical modeling material seen at 107. In the 
configuration shown in Fig. 1, the laser scanner 103 and the image projector 105 would be located 
at the same optical origin to avoid problems of shadowing, occlusion and image distortion. This is 
achieved using a coated mirror 106 transparent to the laser scanner and reflective of the visible 
spectrum, as explained in U.S. Provisional Patent AppHcation Serial No. 60/463,273 filed April 16, 
2003. However, tests have shown that, from a distance of 2 meters and with an operating volume of 
approximately 0.5 x 0.5 x 0.5 m, the maximum extent of the phoxel space occupied by the surface 
of the modeling material 107, a simpler arrangement where the projector and scanner are positioned 
along side each other performs adequately since the scanned and projected rays can be considered 
to originate from the same source. 

[019] The laser scanner 103 determines the geometric position of the surface of the 
modeling material 107 with a high degree of accuracy and allows any opaque non-reflective 
material to be used as a modeling medium. Laser scanners are costly, however, and less accurate 
alternatives may be employed, one of which is described below in connection with Fig. 2. 

[020] The scanner 103 and the image projector 105 should be calibrated to ensure that the 
spatial coordinates of the surface of the material captured by the scanner correspond precisely with 
correct voxel values represented by the image pixels projected onto the surface. The 
scanner/projector pair may both be housed inside a casing (not shown) at a height of approximately 
2 meters above the surface of the modeling material 107. 

[021] The Minolta Vivid 900 used to implement the scanner 103 was designed to perform 
single scans of static objects and scenes. In order to capture changes in the surface geometry of the 
modeling material in real-time, it is controlled by the connected processor to repeatedly scan the 
surface of the material. A scan control routine indicated at 1 1 1 in Fig. 1 is executed by the same 
processor, shown generally at 1 10, which also uses the position data from the scanner delivered at 
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1 12 to select the voxel values in the voxel dataset 114 which may be persistently stored in a hard 
drive or the like connected to the processor 110. This is allows 320 x 240 point values to be 
captured every 0.3 seconds, resulting in a near-real-time capture of the geometry of the upper 
surface of the modeling material 107. This scanned data output at 1 12 is processed to form data 
defining the x, y and z coordinates of the modeling material's surface. As illustrated at 120, this 
surface geometry data is used to identify the those voxel values in a voxel dataset 1 14 which are 
positioned at the intersection of the voxel volimie and the surface geometry established by the 
position of the surface of the modeling material 107. 

[022] As illustrated at 130, the voxel values selected at 120 are then assigned RGB values 
that define the color of each corresponding spot of light projected by the image projector 105 onto 
the three dimensional surface of the modeling material 107. A resulting graphical "phoxel" 
visualization of the voxel dataset values at the surface intersection is accordingly displayed to the 
user who can interactively change the geometry of the surface intersection, and the representation 
of voxel dataset at the surface intersection, by manually deforming the modeling material as 
illustrated at 140. 

[023] The voxel data values being depicted by the displayed image on the surface will 
represent different things in different applications. In medical applications, each voxel value may 
represent the NMR (nuclear magnetic resonance) signal intensity of the contents of the 
corresponding volume element of tissue being analyzed, or may represent the amount of heat being 
given off by a volume element of tissue as measured by a computerized thermal imaging (CTI) 
system. In the aerospace and automotive industries, computational fluid dynamic (CFD) 
simulations may be used to produce voxel datasets that describe the air flow around planes and 
cars. 

[024] Once a voxel dataset has been produced by a sensing or scanning device, or 
computer-generated by a simulation process, or some other means, the resulting original voxel data 
may be pre-processed in a variety of known ways. The pre-processing of a voxel dataset may create 
a modified voxel dataset in order to achieve a variety of objectives, including: (1) defect correction, 
such as noise removal; (2) image enhancement, such as data rescaling or highlighting features of 
interest; (3) segmentation, such as identifying a range of voxel values that separate the features of 
interest from the background; and/or (4) binary image manipulation, such as employing a filter to 
calculate a replacement value for each voxel in the dataset fi-om a combination of its original value 
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and the values of the neighboring voxels, or morphology processing to change the shape of bodies. 
Image data processing is described, for example, in The Image Processing Handbook, by John C. 
Russ, CRC Press (ISBN 0-8493-2516-1). 

[025] The second embodiment of the invention shown in Fig. 2 substitutes an infrared light 
transmission mechanism to determine the position of a manually manipulatable modeling surface 
formed be a bed of glass beads. A box 201 containing a volume of I mm diameter glass beads at 
203 is lit from beneath with an array of 600 high power infrared LEDs as seen at 205. Four IR 
mirrors (not shown) are placed on the interior side walls of the box 201 surrounding the LED array 
to create an infinite plane of uniform radiance. A video projector 208 transmits via the reflective 
glass 209 to the surface of the glass beads at 203. A monochrome infrared camera 210 is mounted 
2 meters above the surface of the beads 203 and captures the intensity of light passing through the 
volume. A voxel dataset 214 is stored in a memory device connected to a processor 215 which 
reconstructs the geometry of the surface of the beads 203 from the image of the surface captured by 
the camera 210 as indicated at 217. The intensity of transmitted light at the surface captured by the 
camera 210 is proportional to the depth of the beads and a lookup table can be used to convert 
surface radiance values into the surface elevation values. The processor 215 then compares the 
geometry data with the voxel dataset 214 as indicated at 220, and also converts the voxel data 
selected at 220 into pixel image values which are delivered to the projector 208. 

[026] The system is calibrated to work with a specific bead size and the optical properties of 
the material used (absorption and scattering coefficients) are critical to its successfiil fimctioning. 
Due to the exponential decay of the IR light passing through the glass beads (or any other material) 
the intensity at the top surface can vary greatly and exceed the optical intensity range of the video 
camera. The solution is to take several images of differing exposure time and combine these to 
recover the effective radiance of the scene, as explained in Recovering High Dynamic Range 
Radiance Maps from Photographs, Debevec and Malik, SIGGRAPH 1997. 

[027] The siu'face geometry measurement scheme of Fig. 2 is less accurate than the use of 
the laser scanner as depicted in Fig. 1. In addition, this approach is material-dependent, requiring 
the use of beads, blocks or the like with predictable light transmission characteristics, and time- 
consuming testing and calibration. However, the system has the advantage of high speed, 
potentially capturing surface geometries at six frames per second, and low cost - an order of 
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magnitude less expensive than the laser scanner. The infrared technique of Fig. 2 provides 
sufficient accuracy for a number of useful appUcations. 

[028] In both configurations, a calibration parameter defining the ratio between the scale of 
the voxel dataset and the scale of the physical modeling surface providing the phoxel-space needs 
to be defined. The system can work at a 1 : 1 scale so that, for example, a physician exploring MRI 
scanning data for the human body can manipulate and visualize three dimensional representations 
of anatomy in their actual size. Li the same way, the system may be used in geophysical 
applications as described below where the scale is 1:5,000,000. The computational operations are 
the same in both configurations: the intersection between the surface of the modeling material and 
the voxel data is calculated and used to determine an image that is projected back onto the three 
dimensional surface of the modeling material 

[029] In both configurations interference with the scanning and projection process may 
result when the user's hands are placed within the scene to manipulate the geometry of the 
modeling material. This problem could be partially eliminated by applying a removal algorithm 
with image-processing techniques or a simple toggle switch. However, in many applications, it is 
preferable to scan the position of the hands and project the hand/voxel intersection data onto the 
user's hands to provide an additional method for exploring voxel datasets. Users can tentatively 
position their hands in phoxel space and use their hands to provide a surface for intersection and 
projection. Once a region of interest is discovered in this way, the modeling material can be used to 
build up a more permanent surface. 

[030] A range of modeling materials can be used to provide an intersection and projection 
surface in phoxel space. Each material has different characteristics and some are more appropriate 
than others for different kinds of data exploration. For example, discrete element models consisting 
of preformed nesting sections are useful for demonstrating predefined regions, such as in the 
teaching of brain anatomy, while a deformable material such as plasticine is useful as means to 
discover the precise shape and location of unsegmented regions, such as the tumor. Modular blocks 
such as sugar cubes can be useful in representing predefined volumes, as in the geophysics example 
described below, and the glass beads are suited to the gradient geometries found in examples like 
the CFD simulation. These materials can be broadly classified by their relative grain size. While the 
larger grain sizes are more suited to predefined, object oriented modeling, smaller grain sizes as 
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found in the glass beads and plasticine offer precise control over form and so are better suited to the 
understanding of unknown, undefined geometries. 

[031] The representation of complex boundaries, such as those found between differing 
tissues, seismic velocity zones, archaeological elements, and equal fluid pressure surfaces, are 
difficult to comprehend using conventional methods. Phoxel space allows this kind of analysis to 
be an easy and intuitive task. In addition, the physicality of the interface also allows multiple users 
to collaborate around the same model, providing a platform for discussion and shared 
understanding. 

[032] The phoxel space interface has been shown to be useful in a number of different 
applications. Three such applications are described below using the laser scanner embodiment of 
Fig. 1, and a fourth employing the injfrared camera with a bed of glass beads seen in Fig. 2. 

[033] 1. Illustrating brain regions using discrete element model. Students are conventionally 
taught about the complex structure of hvunan anatomy through the use of diagrams, photographs, 
models, 3d visualizations, video clips and cadaver vivisections. Each form of representation is 
valuable in a different way. A diagram may clearly depict generalizable properties of the brain but 
may lack the particularity of a photographic image that can record a specific condition. Physical 
models may convey spatial relationships but lack flexibility. Volume renderings may allow 
examination of particular case data, but they rely on being understood as two dimensional visual 
information. Cadaver vivisections allow students to become familiar with the properties of real 
anatomical tissues and spatial relationships but they do not allow for the demonstration of a 
particular condition and it is a destructive process of great expense. Phoxel space can be used to 
represent a regionally segmented model of the human head to be used for teaching and 
demonstration purposes. A physical model of a specific and perhaps rare condition is produced 
from an existing MRI, CTI or cross-sectional dataset using a rapid prototyping process. The 
physical model of the brain is made up of a series of discrete elements that can be removed by 
teachers or students at will. As elements are removed or added, visual information relating to newly 
revealed surfaces is projected onto the surface of the model. Unlike conventional, non-augmented 
models, the phoxel space interface allows a student to switch between MRI, CTI or anatomical 
surface information, and permits dynamic information, such as time varying brain activity, to be 
projected onto the model. 
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[034] 2. Analysis of brain tumor using malleable material. A more demanding application 
uses the phoxel space interface to plan surgical operations in non-regionalized anatomies. While 
visual renderings and MRI cross sections greatly aid in such surgical planning, the phoxel space 
interface allows surgeons to non-invasively explore living anatomical regions in thee dimensions. 
In this example, a malleable material, plasticine, is used to define a freeform cross-section surface 
onto which anatomical MRI voxel data is projected. The surgeon is then able to alter the form of 
this cross section surface by manipulating the plasticine material, revealing the precise spatial 
boundary between, for example, tumor affected regions and normal tissues. This spatial 
understanding would be extremely difficult to convey in any other medium and could greatly 
faciUtate the planning of surgical operations. 

[035] 3. Seismic velocities revealed with block modules. As in biomedicine, the field of 
seismology requires a sophisticated understanding of spatial relationships, in this case within the 
volume of the earth. A phoxel space interface could be used, for example, to explore the seismic 
velocity within the Australian plate, recorded using a network of geophones laid out across the 
continent. The model reaches a depth of 55 km and discrete blocks (sugar cubes) may be used, each 
representing a 5 km depth and a surface area of 50 x 50 km. The geoscientist can add or remove 
these blocks to reveal regions of equal seismic velocity, gaining a clearer understanding of the three 
dimensional composition of the earth's crust. 

[036] 4. CFD simulation represented with continuous material. Phoxel space may also be 
used to gain better xmderstanding of computer-generated datasets, such as those obtained with 
Computational Fluid Dynamic (CFD) simulations. A physical model airplane is half submerged in 
the volume of glass beads. The model is calibrated with three dimensional CFD simulation data 
indicating the air temperature around the airplane geometry. As the glass beads are removed, the 
varying temperatures around the plane form a scaled color map, and the user can manipulate the 
bed of beads to build surfaces which represent regional boundaries or equal temperatures. 

Conclusion 

[037] It is to be understood that the methods and apparatus which have been described 
above are merely illustrative applications of the principles of the invention. Numerous 
modifications may be made by those skilled in the art without departing from the true spirit and 
scope of the invention. 
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